Method of controlling an object by eye motion for electronic device

ABSTRACT

The present invention provides a method of controlling a virtual object or instruction for a computing device comprising: detecting a user eye motion by a detecting device; generating a control signal in responsive to the user eye motion detection; controlling an object displayed on a display in responsive to the control signal to execute the instruction. The user eye motion is detected by CMOS or CCD.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of application Ser. No. 11/889,602,filed Aug. 15, 2007, which is hereby incorporated by reference.

TECHNICAL FIELD

The present invention relates generally to a portable device, and moreparticularly to an electronic device with user motion control module.

BACKGROUND OF THE RELATED ART

Cellular communications systems typically include multiple base stationsfor communicating with mobile stations in various geographicaltransmission areas. Each base station provides an interface between themobile station and a telecommunications network. Mobile telephonesystems are in use or being developed in which the geographic coveragearea of the system is divided into smaller separate cells, itcommunicates with the network via a fixed station located in the cell.Mobile telephones belonging to the system are free to travel from onecell to another. When a subscriber within the same system or within anexternal system wishes to call a mobile subscriber within this system,the network must have information on the actual location of the mobiletelephone.

Recently, the price of cellular telephone has been greatly reduced andbecome affordable to lots of people. It is common that a person ownsmore than one cellular phone. Some people even replace their cellulartelephones as often as they replace their clothes or hairstyle. Thecellular manufactures have to release new models with differentappearances, function and styles more frequently so as to attract theattention of the buyer and occupy a favorable marketing share.Furthermore, the conventional LCD display has a lot of drawbacks, forexample, the LCD display needs back light module which increases thethickness of the panel and it is not transparent. Liquid crystalmaterial and color filter are also essential components to the display.Moreover, the device requires large arrays of thin film transistors, themanufacture is complicated and the resolution is not high enough.Further, the antenna of the conventional device is embedded within thedevice, and shield by a lot of IC or components. The signal reception isinterrupted by a lot of device and EM shielding effect. The mouse isinconvenient to a user, especially, to a portable device. There is adesired to remove the mouse device. As recognized herein, forportability, it is desirable to configure the projector to be as slim aspossible. But the goal of size reduction is frustrated by the present ofthe elements mentioned above. Pluralities of devices, such as mobilephone, PDA, GPS, smart phone, are developed recently. These devicesallow the user communicates with other wirelessly, each of whichutilizes some sort of user interface. The display of the devices havinguser interface typically maybe LCD or other type display. The display istypically configured to present visual information such as text andgraphics. In most cases, the user interface is positioned on the frontface of the electronic device for easy viewing and easy manipulation.However, the trend of the portable device is small, light weight andmulti-function. Typically, the size of the display is limited by thedevice size. It is very inconvenience to browse the web page by thesmall panel. The user should move the scroll of the interface throughtouch pen functional keys or on the key pad. Namely, the user uses thetouch pen or keys moving the web page forwardly and backwardly. Theconventional design is a user-not-friendly design.

SUMMARY

The object of the present invention is to provide a device with hybridmemory and advanced display with thinner transparent panel on which atransparent antenna may be formed to minimize the shielding effect. Thefurther object of the present invention is to provide a computing devicewith eye control module. A method of controlling a virtual object by eyemotion of a user for an electronic computing device having a displayincludes providing at least one image sensor for the display of theelectronic computing device to capture image of the user in front of thedisplay, wherein the at least one image sensor includes CMOS or CCDsensor, wherein the at least one image sensor provided to detect eyemotion of the user to output a first control signal to a virtual objectwithout touching the display, the user being located in front of the atleast one image sensor and keeping the display with a distance;capturing images of the eye motion of the user without touching thedisplay by the at least one image sensor; analyzing image change of theeye motion; generating a first control signal in responsive to the imagechange of the eye motion detection; providing a vocal control through amicrophone and a vocal control software of the electronic computingdevice to output a second control signal through voice recognition; andcontrolling the virtual object displayed on the display in responsive tothe first, second or combination control signal without touching thedisplay of the computing device. The method further provides an infraredray (IR) source for detecting the user. The control signal includescursor movement, character input, application instruction, webpagemovement or image size change. The dual light sources are used toproject two spatially separated spots at user's pupil, and the duallight sources are constructed by side by side on a panel the portabledevice.

A method of controlling a virtual object by eye motion and figure motionof a user for an electronic computing device having a display includesproviding at least one image sensor for the display of the electroniccomputing device to capture image of the user in front of the display,wherein the at least one image sensor includes CMOS or CCD sensor,wherein the at least one image sensor provided to detect eye motion ofthe user to output a first control signal to a virtual object withouttouching the display, the user being located in front of the at leastone image sensor and keeping the display with a distance; capturingimages of the eye motion of the user without touching the display by theat least one image sensor;

analyzing image change of the eye motion; generating a first controlsignal in responsive to the image change of the eye motion detection;capturing images of the figure motion of the user without touching thedisplay by the at least one image sensor; analyzing image change of thefigure motion;generating a second control signal in responsive to the image change ofthe figure motion detection; and controlling the virtual objectdisplayed on the display in responsive to the first, second orcombination control signal without touching the display of the computingdevice.

A method of controlling a virtual object by eye motion and finger motionof a user for an electronic computing device having a display includes:providing at least one image sensor for the display of the electroniccomputing device to capture image of the user in front of the display,wherein the at least one image sensor includes CMOS or CCD sensor,wherein the at least one image sensor provided to detect eye motion ofthe user to output a first control signal to a virtual object withouttouching the display, the user being located in front of the at leastone image sensor and keeping the display with a distance; capturingimages of the eye motion of the user without touching the display by theat least one image sensor; analyzing image change of the eye motion;generating a first control signal in responsive to the image change ofthe eye motion detection; capturing images of the finger motion of theuser without touching the display by the at least one image sensor;analyzing image change of the finger motion;

generating a second control signal in responsive to the image change ofthe finger motion detection; and controlling the virtual objectdisplayed on the display in responsive to the first, second orcombination control signal without touching the display of the computingdevice.

A method of controlling a virtual object motion by user motion for anelectronic device comprises: detecting a user motion by detectingdevice; generating a control signal in responsive to the user motiondetection; and controlling the virtual object displayed on a display inresponsive to the control signal. The user motion is detected by CMOSsensor, CCD, touch sensor array. The user motion includes facial motion,eye motion, or finger motion. The method further comprises a step ofanalysis the user motion before generating the control signal. Theanalysis includes the analysis of image change of the user motion.Alternatively, the present invention discloses a method of instructing avirtual object by user activity for an electronic device comprising:detecting a user activity by a detecting device; generating a controlsignal in responsive to the user activity detection; controlling thevirtual object displayed on a display in responsive to the controlsignal to execute the instruction.

The present invention discloses a computing device comprising: a controlunit and a display; a detecting source for generating a detecting lightto eye of a user; a sensor in responsive to the detecting light backfrom the eye to generate an output signal, thereby generating eyemovement information; a cursor control module in responsive to the eyemovement information to drive a cursor on the display corresponding tothe eye movement information. The computing device further comprises atouch screen module coupled to the control unit to generate at least onebutton-image on the display along with the cursor. Alternatively, thecomputing device further comprises an input interface to allow the userto active “click on” action. The computing device includes but notlimited to notebook, PC, cellular, PDA and the like.

A method for browsing web page, text, email or image on a display of anelectronic device, comprises displaying a first part of the web page,text, email or image on the display; providing sensing array on thedisplay; determining whether or not a user touching the sensing array;determining user motion based on the user's touch; generating a controlsignal based on the user motion; displaying a second part of the webpage, text, email or image on the display based on the control signal,wherein the control signal indicates where the user would like tobrowser. The user motion includes at least one finger motion. The firstpart does not overlap with the second part. Alternatively, the firstpart partially overlaps with the second part.

FIG. 1-2 shows a diagram of a portable device according to the presentinvention.

FIG. 3-4 shows a diagram of a portable device according to the presentinvention.

FIG. 5 shows a diagram of field emitter device according to the presentinvention.

FIG. 6 shows diagram of a EL according to the present invention.

FIG. 7 shows a diagram of dual wireless module according to the presentinvention.

FIG. 8 shows a diagram of digital camera according to the presentinvention.

FIGS. 9, 10, 11 and 12 show a diagram of eye control module according tothe present invention.

FIG. 13 shows a diagram of according to the present invention.

FIG. 14 shows a flow chart according to the present invention.

FIG. 15-FIG. 15-2 show a flow chart according to the present invention.

DETAILED DESCRIPTION

The present invention relates generally to a computing or portabledevice. The device includes but not limited to cellular phone, PDA(personal digital assistant), smart phone, notebook, digital stillcamera, digital video camera, medium player (MP3, MP4), GPS and theequivalent thereof.

FIG. 1 is a diagram illustrating main components of a portablecommunication device using a panel with emitters and transparentsubstrate according to an embodiment of the present invention. Theembodiment, as shown in FIG. 1 and FIG. 2, the device 20 includes a RFmodule 190. As know in the art, the RF module 190 includes antenna. Thisantenna is connected to a transceiver, which is used to receive andtransmit signal. AS known, the RF module 190 further includes CODEC, DSPand A/D converter as well. Due to the RF module is not the feature ofthe present invention, therefore, the detailed description is omitted.Other major components between device 10 and 20 are similar, therefore,the same reference numbers refers to similar component, however, theversion, grade and performance maybe different. The present inventionincludes a central control IC 100, an input and output (I/O) unit 150,OS 145, hybrid memory 165, the device 10 or 20 may includes other memory155 such as ROM, RAM and FLASH memory. The RF module may perform thefunction of signal transmitting and receiving, frequency synthesizing,base-band processing and digital signal processing. If the portabledevice is cellular, SIM card hardware interface is provided forreceiving a SIM card. Finally, the signal is send to the finalactuators, i.e. a loudspeaker and a microphone 195 or I/O 150.

The present invention further includes a wireless transmission/receivingmodule 220 coupled to the control IC 100. The transmission/receivingmodule is compatible with blue-tooth, home-RF, 802.11x, WiFi, WiMAXstandard or their higher version. The transmission domain (the air) bynature is not secured and therefore encryption maybe essential in thewireless transport networks. In one embodiment, pair-wiseencryption/decryption between every neighboring wireless network deviceof a wireless transport network is well-known in the art. A data framethat leaves from one wireless device from one end of a wirelesstransport network to the other end of the same network might needseveral encryptions and decryptions before it reaches its finaldestination. A mini projection module 230 maybe incorporated into theportable device, the mini projection module 230 may refer to theco-pending application Ser. No. 11/734,175, filed by the same inventorof the application.

Optionally, the present invention also provides a hard disk driveincorporated with a Flash memory within one single unit, as shown inFIG. 3 to have the benefits of both. The hard disk drive includes acontrol unit 400 coupled to a magnetic disk module having spindledriver, reading/writing head mechanism and so on. This is well-known inthe art, the description is omitted. A flash controller 440 is coupledto the control unit 400 and a nonvolatile memory 430 is coupled to theflash controller 440. An application 450 is coupled to the control unitto execute the instruction. The hybrid hard disc drive incorporates theflash memory into the device to yield the benefit of high-speed accessand low power consumption. The hard disk drive maybe incorporated intoportable media player, cellular, notebook or PDA. A controller 440 isconnected to the flash memory 430 to operate the mode under instructionor command. The operation mode includes read mode, write mode, erasemode, and programming mode. Further, the controller 440 maybe integratedinto the control unit 400 under the knowledge and technology ofintegrated circuits. An embedded system 470 is optionally stored in theerasable/programmable storage medium (flash, ROM or hard disk). It couldbe an application, software or a list of computer-executableinstructions. Alternatively, the ROM 480 may be embedded in the controlunit. The data coded in the ROM will not be erased by the user. Theembedded system 470 could be provided for a computer or device toexecute predetermined function such as operating some application,booting the device, auto backup or auto-run file. In general, after adevice is power on, basic input/output system will perform a bootsequence before invoking the OS. Basic input/output system is set in theROM. The programmer or instruction can be set in the embedded system todetermine what type of data will be stored into what type of storage(flash or hard disk). It could be used for different type of volatilememory having NOR device and NAND device (FIG. 4). A hybrid memoryscheme may refer to the above co-pending application filed by theidentical inventor of the application, entitled “Monitoring System witha Wireless Transmitting/Receiving Module”. The co-pending application isincorporated herein for reference. Under such configuration, thecomputing device, for example, personal computer, notebook, may omit thehard disc drive. The scheme uses NOR device to act and replace the roleof hard disc to store operation system and yield the benefit of highspeed operation and low power consumption, while the data is stored inthe NAND device. The operation speed will be higher than the oneemploying hard disc. In one embodiment, the embedded system is coupledto the control unit to determine or assign the duty of the magnetic diskmodule 420 and the nonvolatile memory 430. The flash benefits high speedperformance, low power consumption while hard disk drive yields lowercost and high capacity. The storage architecture refers to ComplementaryStorage Device (CSD). The CSD has the benefits of the both type ofmemory. The manufacture may reduce the cost and increase the performanceby incorporating the two kind of memory with an optimized memory ratiobetween the flash memory and the magnetic memory.

Almost all conventional devices include an antenna located within theportable device, the signal receiving performance is poor due to EMshielding effect generated by the shield, circuits, circuit board andother components. If the antenna to the signal is “transparency”, theperformance will be improved. Therefore, in another aspect of thepresent invention is to provide an antenna located corresponding to asubstantially transparent panel to minimize the EM shielding effect,thereby improving signal receiving/transmitting performance. Preferably,at least one transparent antenna is attached on the substantiallytransparent panel to minimize the EM shielding effect. As seen in FIGS.1 and 2, a substrate 400 is provided and transparent electrodes 420 areformed on the glass substrate 400. The substrate 400 could be silicon,glass, quartz or the like. The embodiment of the present invention isshown in FIG. 5, it is a cross-sectional view of the field emissiondevice according to the present invention. As seen in FIG. 5, atransparent substrate 400 is provided and transparent electrodes 420 areformed on the glass substrate 400. The transparent electrodes 420 may bemade of indium tin oxide (ITO) and may be used as the emitterelectrodes. Stacked gate 410 that cover a portion of the transparentelectrodes 420 are formed on the glass substrate 400. Emitters 460 thatemit electrons are formed on a portion of the transparent electrode 420.Each stacked gate 410 includes a mask layer 440 that covers a portion ofthe transparent electrodes, and is formed by UV photolithograph mask.The mask layer 440 is preferably transparent to visible light, butopaque to ultra violet rays and can be made of an amorphous siliconlayer. The silicon layer will be transparent when the thickness is thinenough. A stacked gate 410 structure includes first insulating layer/agate electrode/a second insulating layer/focus gate electrode,sequentially formed over the substrate. The gate insulating layer ispreferably a silicon oxide thin film with a thickness of 2 mu.m or moreand the gate electrode is made of chrome with a thickness of about 0.25.mu.m. The gate electrode is used for extracting an electron beam fromthe emitter. The focus gate electrode performs as a collector forcollecting electrons emitted from emitter so that the electrons canreach a fluorescent film 480 disposed above the emitter 460. If thedevice is used for display, the substrate can be silicon or transparentsubstrate. Referring to FIG. 5, a front panel (substrate) 450 isdisposed upward and above the stacked gate. A variety of visual imagesare displayed on the front panel 450. A fluorescent film 480 is attachedto a bottom surface of the front panel 450 that faces the stacked gateand a direct current voltage is applied to the fluorescent film 480 toemit color for display. The fluorescent substance may emit color lightby mixing the emitted light if the thin film with R, G, B fluorescentsubstances. Preferably, the fluorescent substances emit red, green, andblue visible light when excited by the electron beam is evenlydistributed on the fluorescent film 480. Spacer separating the frontpanel 450 from the stacked gate is a black matrix layer and is not shownfor convenience. Due to the thin film display if formed with thinnerthick and the power consumption is lower than LCD, the present inventionmay provide smaller size, lighter weight device. The life time ofbattery may last longer. The field emission device does not requirecomplicated, power-consuming back lights and filters which are necessaryfor LCD. Moreover, the device does not require large arrays of thin filmtransistors, and thus, a major source of high cost and yield problemsfor active matrix LCDs is eliminated. The resolution of the display canbe improved by using a focus grid to collimate electrons drawn from themicrotips. Preferably, the emitter includes a carbon nanotube emitter tofurther reducing the device size. Further, the display may omit theliquid crystal material. Further, the field emission display does notrequire the S/D regions which are required by TFT for LCD.

In another embodiment, the display is shown in FIG. 6. The displayincludes a transparent electrode 510 on a transparent substrate 500. Afluorescent film or power 520 is attached to an upper surface of thelower transparent electrode 510. Preferably, the fluorescent substanceemits color light. The present invention includes three such devicesthat separately display image in red components, green components, andblue component. Each irradiates single color light. Different powderwill emit different color. An upper transparent electrode 530 is formedon the fluorescent film or power 520. A second transparent substrate 540is formed on the transparent electrode 540. A bias is applied on theelectrodes to inject hole and electron, thereby exciting the fluorescentsubstances by the combination of the electron and hole to emit red,green, or blue visible light depending on the compound of thefluorescent substances. The elements may refer to ELP.

In another embodiment, the wireless data transferring module 220includes dual mode module. Please refer to FIG. 7, in one case, thewireless data transferring module 220 includes a first and secondwireless module 600 a and 600 b for wireless transmission. The dualmodules 600 a and 600 b are coupled to a management unit 600 to managethe dual modules according to the policy set in a policy engine 610. Forexample, the policy in the policy engine 610 includes at least thetransmission priority policy to determine which module will be thedefault module to receive/transmit data. It also includes the switchingcondition for switching there between. For example, the signal strengthis one of the facts for switch condition. It also allows the user to setor alter the condition via user interface. The first and second wirelessmodule maybe one of the following module: blue-tooth, 802.11x, WiFi,WiMAX, 3G standard or their higher (update) version. Preferably, thefirst wireless module is WiFi and the second wireless module is WiMax.The present invention may yield the benefits of both. For example, theaccess range of WiFi is shorter than WiMax, but it consumes lower power.If within the office area, the AP for WiFi may be employed to preventothers outside the office from accessing the server of the office. Inanother aspect, if the user would like to access or remote control aterminal device located at several miles away, the WiMax is chosen. TheWiFi benefits security, low power consumption while WiMax yields longrange and high bandwidth. The module architecture refers to dualwireless module (DWM). The DWM has the benefits of the both type ofmodule. The first wireless module is compatible to a first communicationprotocol, while the second one is compatible to a second communicationprotocol. The manufacture may increase the performance by incorporatingthe two kind of wireless module memory with an optimized configuration.The protocol maybe adapted to wireless local area network or wirelessmediate area network.

From above, the thick and complicated LCD panel is removed, therebyreducing the thickness of the portable device and a lot of elements suchas color filter, LC material, back light module are also skipped fromthe present invention. Furthermore, due to the substrate 400, 450 couldbe formed by glass, quartz or the like, therefore, the present inventionmay offer unexpected and funny visual effect. Under the sleep mode oroff state of the device, the panel is substantial transparent, while thedevice is not operation or standby, or at the sleep mode, the panel mayacts as digital photo album due to the panel is transparent. Further,the user may alter the color, image pattern displayed on the panel whenthe device is in the sleep mode or off-state. The user may determine thepattern and image for the panel cover. Furthermore, a transparentantenna may be attached on the transparent panel to provide bettersignal transmission due to EM shield effect of the antenna will beminimized while the antenna is not set within the device. In this case,the antenna is composed by a material includes oxide containing metal oralloy, wherein the metal is preferable to select one or more metals fromAu, Zn, Ag, Pd, Pt, Rh, Ru, Cu, Fe, Ni, Co, Sn, Ti, In, Al, Ta, Ga, Geand Sb. Some of the transparent material includes oxide containing Znwith Al2O3 doped therein.

Further, referring to FIG. 8, the wireless transferring module 220 andhybrid memory maybe employed by a digital camera or digital videocamera. Preferably, the device includes a main body having a control IC802; a display 804 formed on the main body and coupled to the control IC802; an image capture element 806 formed within the main body andcoupled to the control IC 802; the hybrid memory 808 is coupled to theprocessor; a lens mechanism 810 is formed on the main body, coupled tothe control IC 802 and corresponding to the image capture element 806;the wireless transferring module is coupled to the processor for datatransferring with external device.

If the present invention is employed for medium player such as MP3player, MP4 player, the player includes an analog/digital (A/D)converter for converting analog audio signals into digital audiosignals. The analog audio signals can come from an audio source coupledto player. A digital signal processor (DSP) or an audio and/or videodriving module, for instance MP3, MP4 codec, are coupled to A/Dconverter to receive the digital audio signals. In one embodiment, MP3or MP4 codec executes a firmware that includes a MPEG audio layer (e.g.,MP3, MP2, or both) codec or video codec (e.g., MP4), and DSP executes afirmware that includes a different type of audio codec (e.g., WMA, ACC,or both). In one embodiment, the firmware for DSP also includes a videocodec for encoding and decoding videos (e.g., MPEG-4 V1/V2/V3, DivX3.11/4.0/5.0, Xvid, AVI/ASF, or any combination thereof). MP3 (or MP4)codec and DSP are coupled to a nonvolatile memory that stores thecompressed audio data. The user can select an audio file fromnonvolatile memory. DSP are coupled to an audio processor, whichprocesses the digital audio signals according to default settings oruser instructions. Audio processor is coupled to a digital/analog (D/A)converter, which converts the digital audio signals into analog audiosignals for the user.

The present invention also provides a user control module to control thecursor without mouse or touchpad. A computing device comprises a displayand a detecting device on said display for detecting motion of a user. Amovement information generating device is in responsive to the detectionto generate an output signal, thereby generating movement information. Acursor control module is in responsive to the movement information todrive a cursor on the display corresponding to the movement information.Referring now to the drawings 1-2 and 9, there is shown in schematicform the basic components of the control module 185 incorporating theeye, finger or face control module according to a preferred embodimentof the invention. The present invention includes a step of detecting thefinger motion of a user. For example, in FIGS. 9 and 10, the user islocated in front of the sensor in order to detect the motion of finger.Preferably, the portion for detection could be eye, finger, face or thelike. The eye detection will be introduced as one of the examples toillustrate the features of present invention. The subject's face,finger, or eye is positioned relative to a sensor so that initially thesubject's gaze is aligned along center line toward a pupil stimulus andfixation target. The eye control module 185 includes sensor and IC todetect eye motion and generate a control signal. The face motion couldbe used to practice the present invention. A detecting source 18505 isprovided, the pupil of the eye(s) is (are) illuminated by the lightsource 18505, for example, an infrared ray (IR) or light emitting diode(LED). The IR may be used for distance measurement. Preferably, dualsource LED is used to project two spatially separated spots at thesubject's pupil. The dual source LED is constructed by placing two LEDside by side on the panel 400 a of the portable device. Back light fromthe subject's eye is detected by a sensor 18510 directly or via otheroptical mirror or lens. Another method is to detect the user face motionor image by the sensor. The sensor 18510 could be optical sensor such asCMOS sensor or CCD. The outputs from the sensor 18510 are input to aprocessor or control integrated circuits 18515 to generate a controlsignal to a cursor control module 18520 for controlling a cursor on thedisplay or panel. Eye detection method may refer to U.S. Pat. Nos.7,130,447 and 5,943,754, both are assigned to The Boeing Company(Chicago, Ill.); U.S. Pat. No. 5,670,613, assigned to Pulse MedicalInstruments, Inc. (Rockville, Md.). U.S. Pat. No. 5,422,690 discloseseye recognition and blink detection system. Preferably, the detectingsource or the like scans the position of the pupil of eye(s). In thisprocess the pupil is illuminated by a light source, so that thegeometric form of the pupil can be portrayed clearly on the sensor.Alternatively, the image (face, finger) change of the user could bedetected by the present invention. By means of image processing, thepupil position information is evaluated and to determine where the eyein the display is looking. The control signal may drive the cursor tothe position where the eyes are looking through cursor control module18520. A buttons-image (or button-icons) may be generated along with thecursor by an image generator 18525. In one case, the image generator18525 maybe a touch screen module which may generate touch screen imagevia well-known touch screen technology, in the manner, the user may“click on” the virtual bottom to input a command by means of “clicking”the touch screen. Alternatively, the click signal maybe input from inputinterface 18540 such as (the right and left buttons of) the keypad,vocal control through microphone, eye motion through the sensor 18510.In the case of vocal control, another software/hardware maybe necessaryto process the steps of object selection through voice recognitionhardware and/or software. For example, the action of close left eyerefers to click left button while the action of close right eye refersto click right button. If both eyes close, it may refer to select oneitem from a list. The above default function may be practiced by aprogram and software. It should be understood by persons skilled in theart, the foregoing preferred embodiment of the present invention isillustrative of the present invention rather than limiting the presentinvention. Modification will now suggest itself to those skilled in theart. Under the method disclosed by the present invention, the user maymove the cursor automatically without the mouse. Similarly, the controlsignal may be used to drive the scroll bar moving upwardly or downwardlywithout clicking the bar while reading document displayed on the screen,as shown in FIG. 11. Thus, the control signal generated by IC will befed into the scroll bar control module 18550 to drive the scroll bar18555 on the display moving upwardly or downwardly without the mouse orkeypad. An eye controllable screen pointer is provided. The eye trackingsignals are performed in a calculation by a processing means residing ina processor or integrated circuits to produce a cursor on the screen.

The sensor is electrically coupled to the controller (IC) 18515 vialine. In a preferred embodiment, input controller 18515 comprises asemiconductor integrated circuit or chip configured to receive,interpret and process electrical signals, and to provide outputelectrical signals. Output signals from IC 18515 comprise signalsindicative of movement of eye in a direction corresponding to thedirection of actual cursor movement on the display intended by the user.The present embodiment takes into account a possible “dragging”situation that the user may be faced with. On occasion, some users havea need to “drag” an icon or other object from one area of the screen toanother. On some computers, to accomplish this, the user must hold downthe left click button and control the pointing device at the same time.If a touchpad is being used as the pointing device, and the object musta dragged a long distance across the screen, sometimes the user's fingermay reach the edge of the touchpad. This situation is easily handled bythe present invention. In such a situation, the controller may send thecommand (e.g. “click left mouse button”, while dragging) repeatedlyuntil the user's finger leaves a keyboard key (stops pressing a key).This permits dragging to be performed even after the user's fingerleaves the touchpad. U.S. Pat. No. 7,165,225, assigned to MicrosoftCorporation (Redmond, Wash.) disclosed “Methods and systems for cursortracking in a multilevel GUI hierarchy”. U.S. Pat. No. 7,095,447,assigned to Samsung Electronics Co., Ltd. (Kyonggi-do, KR) disclosed“OSD (on screen display) cursor display method and cursor image displayapparatus”. U.S. Pat. No. 7,137,068, assigned to Microsoft Corporation(Redmond, Wash.) disclosed “Apparatus and method for automaticallypositioning a cursor on a control”. U.S. Pat. No. 6,774,886, assigned toMatsushita Electric Industrial Co., Ltd. (Osaka, JP) disclosed “Displaysystem, cursor position indication method, and recording medium”.

Therefore, the present invention providing a method of pointing a marksuch as cursor, bar on a screen, the method includes detecting motion ofa user (such as eye, face motion) and a sensor is in responsive to thedetection of the eye to generate an output signal, thereby generatingeye movement information; A cursor control module is in responsive tothe user movement information to drive a cursor on the displaycorresponding to the movement information.

Similarly, the above method maybe used for face tracing in the field ofdigital still camera or digital video camera to tracking the face of thesubject. By the almost same scheme, a face indication (or mark) module18545 is responsive to the control signal to mark the face on thescreen, thereby tracking the face for the digital camera. A digitalcamera comprises a control unit and a display; a detecting source fordetecting eye of a user who is under photographed; a sensor inresponsive to the detecting light back from the eye to generate anoutput signal, thereby generating eye movement information; a cursorcontrol module in responsive to the eye movement information to drive aface indicator on the display corresponding to the eye movementinformation. The digital camera further comprises a wireless datatransferring module coupled to the control unit for data transferringwith an external device.

As aforementioned, the present invention discloses a user motion controlmodule for computer, display or portable device. The module could beincorporated into the device adjacent to the keypad, display or keyboardarea. Then, it may detect the finger or figure motion of the user tomove the cursor or virtual objects. Under some embodiments, the CMOS orCCD is used to detect the user motion including the facial expression,facial motion, figure motion or finger motion. In these applications,the sensor may capture the images of the user's finger or figure motionand the controller may analysis the image change, thereby determiningthe cursor movement. The monitoring of and response to the user's facialexpressions may also be used, for example, the user's motion could bemonitored with a still camera or a video camera. As aforementioned in[0030], a detecting source 18505 is provided and the user is illuminatedby the light source 18505, for example, an infrared ray (IR) may be usedfor distance measurement. For example, in FIGS. 9 and 10, in order todetect the motion of user's finger or figure, the user is located infront of the sensor and the display. It is unlike the conventional trackball, control panel for notebook. The sensitivity, resolution andcontrollability of the control panel are not so good. It should benoted, in the embodiment, the user motion detecting module is setadjacent to the keypad of notebook, display or keyboard of the PC. Theuser motion detecting module detects the finger or figure motion of theuser by CMOS, CCD as aforementioned method. The resolution of the CMOSsensor may achieve higher than several Mega pixels. It may preciselyreflect the finger or figure (or face) motion of the user.

Alternatively, the cursor or items or function of computer (such as openfile, close file, copy, cut, paste, etc.) may be controlled by the useractivity, such as through the measurement of the activity of the humanbrain. The EEG (electroencephalograph) records the voltage fluctuationsof the brain which can be detected using electrodes attached to thescalp. The EEG signals arise from the cerebral cortex, a layer of highlyconvoluted neuronal tissue several centimeters thick. Alpha waves (8-13Hz) that can be effected if the user concentrates on simple mentallyisolated actions like closing one's eyes; Beta waves (14-30 Hz)associated with an alert state of mind; Theta waves (4-7 Hz) usuallyassociated with the beginning of sleep state by frustration ordisappointment; and Delta waves (below 3.5 Hz) associated with deepsleep. Electromyographic (EMG) sensors are attached to the person's skinto sense and translate muscular impulses. Also Electrooculargraphic(EOG) signals have been sensed from eye movement. U.S. Pat. No.7,153,279, assigned to George Washington University disclosed a brainretraction sensor. U.S. Pat. No. 7,171,262, assigned to Nihon KohdenCorporation disclosed a Vital sign display monitor. FIG. 12 is a diagramof an illustrative embodiment of the invention. The neural activity istracked on neural activity detecting device 350. Preferably, the neuralactivity tracked includes EEG, EOG, EMG activity. The electrical signalsrepresentative of the neural activity are transmitted via wired orwireless to the control unit. If a predetermined signal is sensed bydetecting device 350, the same EEG readings may be monitored. Forexample, the Alpha waves (8-13 Hz) can be effected if the userconcentrates on some actions. Thus, if the concentration pattern isdetected, the system is responsive to the signal and issue aninstruction to take action to “open file”, “close file”, “copy file”,“clicking”, “paste”, “delete”, “space”, or “inputting characteristics”etc. It should be noted that the state patterns of potential users maybe monitored before the system is used.

The control IC 18515 is coupled to a signal receiver (not shown) whichreceives the neural signals from sensor 18510 by antenna or wired. Anoperating system runs on CPU, provides control and is used to coordinatethe function of the various components of system and Applicationprograms 18560. These programs include the programs for converting thereceived neural electrical signals into computer actions on the screenof display. By using the aforementioned devices, a user is capable ofcontrolling the computer action by inputting neural information to thesystem through sensor. There will be described the setting up of aprogram according to the present invention for a user controlling acomputer with sensed neural signals. A program is set up in the computerto use the electrical signals to control computer functions and/orfunctions controlled by the computer. A process is provided forpredetermining the neural activity level (or pattern) that indicates thelevel of concentration of the user. A sensor is provided for monitoringa user's neural activity to determine when the predetermined neuralactivity level has been reached. The user's EEG pattern is determined.The user's neural activity is converted to electrical signals, and togive an instruction to execute a software functions. Before the user EGGpattern is determined, an image sensor (CCD or CMOS) is introduced tomonitor the facial motion (or eye motion) to determine where the userlooks at on the screen.

Therefore, the present invention discloses a method of controlling acursor by user motion for a computing device comprising without touchthe display: detecting a user motion by detecting device; generating acontrol signal in responsive to the user motion detection; andcontrolling the cursor displayed on a display in responsive to thecontrol signal. The user motion is detected by CMOS or CCD and the usermotion includes facial motion, eye motion, or finger motion. The methodfurther comprises a step of analysis the user finger motion beforegenerating the control signal. The analysis includes the analysis ofimage change of the user finger (or figure) motion. The frame rate ofthe CMOS may achieve 30 FPS (frame per second). Thus, during 0.5 second,at least 15 frames are taken; the present invention has enough data toanalysis the finger image and may analysis the finger image changeeasier. During the time interval, only the fingers are moving, and thebackground is stand still relative to the fingers. After analysisseveral frames of the finger images, it may determine that the fingermotion direction is from, for example, down-left to up-right, therebycontrolling the virtual object. The sensor is provided to detect a userin front of the display and the sensor.

A method of instructing a virtual object by user activity, for instance,figure or figure motion for a computing device comprises detecting auser activity by a detecting device; generating a control signal inresponsive to the user activity detection; controlling the virtualobject displayed on a display in responsive to the control signal toexecute the instruction. The user activity is detected by CMOS or CCDand the user activity includes facial motion, eye motion, figure orfinger motion. The analysis includes the analysis of figure or fingerimage change of the user. Alternatively, the user activity is detectedby EEG, EMG, or EOG sensor. The control signal includes cursor movement,character input, software application instruction. The virtual objectincludes cursor, control bar, icon, displayed image, web page. In thecase of figure detection, the user may keep a distance to the CMOS orCCD for remote control. The user is located in front of the image sensorand keeps the display with a distance measure by the detecting lightsource 18505 for remote control and the user's hand is not necessaryover a keyboard and the virtual object is control by the user figuremotion without touching the display.

A method of instructing an object by user activity for a computingdevice comprises detecting a user motion by a detecting device by CMOSor CCD; generating a control signal in responsive to the user motiondetection; controlling the object displayed on a display in responsiveto the control signal; detecting a EEG, EMG, EOG pattern by a EEG, EMG,EOG sensor to execute an instruction.

FIG. 13 is a simplified diagram of a portable electronic device 1310, inaccordance with one embodiment of the present invention. The portableelectronic device 1310 may for example be a hand held electronic devicesuch as cellular phones, PDAs, media players, and GPS, or notebook,Tablet PCs and game players. The portable electronic device 1310 isconfigured with a sensor array on the display. The sensor array 1320 isconfigured to detect the presence of an object such as a finger as wellas the location and pressure being exerted on the surface of the panelby the finger or palm of the hand. By way of example, the sensor array1320 may be based on capacitive sensing, resistive sensing, surfaceacoustic wave sensing, thermal sensing and/or the like. The sensor array1320 may further be based on pressure sensing such as strain gauges,force sensitive resisters, load cells, pressure plates, piezoelectrictransducers or the like.

As shown in FIG. 13, the portable electronic device 1310 includes ahousing and a display 1330 situated in a front surface of the housing.The portable electronic device 1310 also includes a touch sensing device1320 is situated on the display. FIG. 13 is a perspective diagram of ahand held electronic device 1310, in accordance with one embodiment ofthe present invention. The hand held electronic device 1310 includes ahousing that encloses internally various electrical components includingintegrated circuit chips. By way of example, the housing may contain amicroprocessor (e.g., CPU), memory (ROM, RAM), a power supply (e.g.,battery), a printed circuit board (PCB), a hard drive or other memory(e.g., flash) and/or various input/output (I/O) support circuitry. Thehand held electronic device 1310 also includes a display 1330 disposedwithin and viewable through an opening in the housing. The display 1330is typically placed on the front surface of the device 1310. The display1330 provides visual information in the form of text, characters orgraphics. By way of example, the display 1330 may correspond to a liquidcrystal display (LCD), organic light emitting diodes (OLED), or adisplay that is based on electronic inks, electronic paper.

In order to generate user inputs, the hand held electronic device 1310may include a sensing array 1320 that is a transparent input panelpositioned in front of the display 1330. The sensing array 1320generates input signals when an object such as a finger is moved acrossthe surface of the sensing array 1320, for example linearly, radially,rotary, etc., from an object holding a particular position on the array1320 and/or by a finger tapping on the array 1320. In most cases, thesensing array allows a user to initiate movements in a GUI by simplytouching the display screen via a finger. For example, the sensing array1320 recognizes the touch and position of the touch on the display 1330and an interpreting controller 1340 of the hand held electronic device1310 interprets the touch and thereafter performs an action based on thetouch event. In accordance with one embodiment, the sensing array 1320is a multi-touch sensing device that has the ability to sense multiplepoints of contact at the same time and report the multiple touches tothe controller of the handheld electronic device. In one implementation,the sensing array 1320 is a multipoint capacitive touch screen that isdivided into several independent and spatially distinct sensing points,nodes or regions that are positioned throughout the display. The sensingpoints, which are typically transparent, are dispersed about the sensingarray with each sensing point representing a different position on thesurface of the display. The sensing points may be positioned in a gridor a pixel array where each pixilated sensing point is capable ofgenerating a signal. The signal is produced each time an object ispositioned over a sensing point. When an object is placed over multiplesensing points or when the object is moved between or over multiplesensing point, multiple signals can be generated. The sensing pointsgenerally map the touch screen plane into a coordinate system such as aCartesian coordinate system or a Polar coordinate system. An example ofa multipoint capacitive touch screen may be found in U.S. patent Ser.No. 10/840,862, which is herein incorporated by reference.

The hand held electronic device 1310 may be designed to recognizegestures applied to the sensing array 1320 and to control aspects of thehand held electronic device 1310 based on the gestures. The gestures maybe made through various particularly finger motions. The hand heldelectronic device 1310 may include a gesture operational program, whichmay be part of the operating system or a separate application. Thegestural operation program generally includes a set of instructions thatrecognizes the occurrence of gestures and informs one or more softwareagents of the gestures and/or what action(s) to take in response to thegestures. By way of example, gestures that can be used are disclosed ingreater detail in U.S. patent Ser. No. 10/903,964, which is hereinincorporated by reference.

In one embodiment, the sensing input device is mapped to the display.When mapped, points on the sensing input device coincide with points onthe display, i.e., have the same coordinates (x and y). Therefore, whena user touches the sensing input device surface, it will appear as ifthe user is touching the image at the same location of the display. Asshown, the sensing array 1320 is divided into several independent andspatially distinct sensing points (or regions) that are positionedwithin the respective component. The sensing points are generallydispersed about the respective component with each sensing pointrepresenting a different position on the surface of the component andthus the device 10. The sensing points may be positioned in a grid or apixel array where each pixilated sensing point is capable of generatinga signal. The number and configuration of the sensing points may bewidely varied. The number and configuration of sensing points generallydepends on the desired resolution of the touch sensitive surface and maybe varied from spot to spot across the display to achieve any desiredcompromise between cost and functionality. In the case, a signal isproduced each time the finger is positioned over a sensing point. Whenan object is placed over multiple sensing points or when the object ismoved between or over multiple sensing points, multiple position signalsare generated. As should be appreciated, the number, combination andfrequency of signals in a given time frame may indicate size, location,direction, speed, acceleration and the pressure of the finger or palm onthe surface of the device. By way of example, the control system may bea microcontroller located within the housing of the device 1310.

The signals generated at the sensing points may be used to determine howthe user would like to move the web page displayed on the display. Byway of example, each portion of the hand in contact with the deviceproduces a contact patch area. Each of the contact patch areas coversseveral sensing points thus generating several signals. The signals maybe grouped together to form a signal that represents how the user ismoving the web page. In one embodiment, the difference between a currentsignal and a last hand signal may indicate the user's desire toimplement a function of moving web-page. A significant differenceindicates the user's desire to implement a function. Changes betweencontact patch areas may further indicate the particular moving signal.In mapping, the touch surface is divided into one or more button zonesthat represent regions of the device that when selected implement theparticular button function associated with the button zone. The buttonzone having the contact patch area with the most significant changebetween first and second hand signals is the one that is typicallyimplemented. The position and size of the button zones may also becustomizable. For example, page back, page next and so on. Thecustomization may be performed by the user and/or the device. It isbecause that the display is to small, the whole web page (or text,image) can not be seen and displayed by the display, as shown in FIG.15. The display of PC may browser the almost the full page of the webinformation, however, it can not achieve the purpose by the portabledevice with small display. From FIG. 15, the user may only browser apart of the “actual web page”, not full page of the web information dueto the limitation of the size of the display. The other area outside theactual display window indicated in the FIG. 15 can not be viewed by theuser. Conventionally, the user should click the scroll bar or keys toscroll the web page (or text, image, email) back and forth, right andleft. It is very inconvenient to the user and the convention method isnot user friendly design. However, the present invention provides thesolution. The user may move his finger on the screen to introduce a user(or finger) movement for indicating where the user would like to view.For example, the user may move his finger and the movement is indicatedby the arrow from area A to area C through area B to browse the part(area C) of the web page, as shown in FIG. 15-1. Thus, the user may movehis figure on the display to review anywhere of the web page (image ortext) while the display is too small, especially for the portabledevice, as shown in FIGS. 15-1 to 15-2.

FIG. 14 is an operational method in accordance with one embodiment ofthe present invention. The method 1420 generally begins at block 1422where the device is in standby. Standby generally implies that thedevice is in a state of readiness waiting for something to happen, i.e.,a user initiating an action therewith. Following block 1422, the processflow proceeds to block 1424 where a determination is made as to whetherthe user is touching the device. This is generally accomplished withtouch sensing device capable of generating signals when a hand nears thedevice and a control system configured to monitor the activity of thetouch sensing device. If it is determined that the user is not touchingthe device, then the process flow proceeds back to block 1422 therebykeeping the device in standby. If it is determined that the user istouching the device, then the process flow proceeds to block 1426 wherethe user is determined.

In one embodiment, once the second location is determined, the processflow proceeds to block, at least two sensing points signals are detectedby the controller. Following block 1428 the process flow proceeds toblock 1430, where touch events are monitored, control signals aregenerated based on the touch event. The control signals 1432 may be usedto inform the application software within the device to move the webpage displayed on the screen instead of by moving the web page by scrollusing keys, cursor or touch pen.

The processor can be implemented on a single-chip, multiple chips ormultiple electrical components. For example, various architectures canbe used for the processor, including dedicated or embedded processor,single purpose processor, controller, ASIC, and so forth. In most cases,the processor together with an operating system operates to executecomputer code and produce and use data. The operating system maycorrespond to well known operating systems such as OS/2, DOS, Unix,Linux, and Palm OS. Memory provides a place to store computer code, thememory may include Read-Only Memory (ROM), Random-Access Memory (RAM),hard disk drive, flash memory and/or the like. The display is generallyconfigured to display a graphical user interface (GUI) that provides aneasy to use interface between a user of the electronic device and theoperating system or application running thereon. The electronic devicealso includes a touch screen that is operatively coupled to theprocessor. The touch screen is configured to transfer data from theoutside into the device. The electronic device also includes a sensingdevice that is operatively coupled to the processor. The sensing devicemay also be used to issue web page moving commands.

Examples of hand held devices include PDAs, Cellular Phones (smartphone), Media player, Game players, Cameras, GPS receivers, notebook,tablet, digital camera, and the like. Therefore, the user may move theweb page, image or document displayed on the page by directly moving thefinger on the sensing array. The user may move the web-page, text,image, icon shown on the display directly by hand or user finger.Alternatively, the aforementioned embodiment may be used, independentlyor combination, to control or switch the TV channel, virtual object,volume and so on.

As will be understood by persons skilled in the art, the foregoingpreferred embodiment of the present invention is illustrative of thepresent invention rather than limiting the present invention. Havingdescribed the invention in connection with a preferred embodiment,modification will now suggest itself to those skilled in the art. Thus,the invention is not to be limited to this embodiment, but rather theinvention is intended to cover various modifications and similararrangements included within the spirit and scope of the appendedclaims, the scope of which should be accorded the broadestinterpretation so as to encompass all such modifications and similarstructures. While the preferred embodiment of the invention has beenillustrated and described, it will be appreciated that various changescan be made therein without departing from the spirit and scope of theinvention.

I claim:
 1. A method of controlling a virtual object by eye motion of auser for an electronic computing device having a display comprising:providing at least one image sensor for said display of said electroniccomputing device to capture image of said user in front of said display,wherein said at least one image sensor includes CMOS or CCD sensor,wherein said at least one image sensor provided to detect eye motion ofsaid user to output a first control signal to a virtual object withouttouching said display, said user being located in front of said at leastone image sensor and keeping said display with a distance; capturingimages of said eye motion of said user without touching said display bysaid at least one image sensor; analyzing image change of said eyemotion; generating a first control signal in responsive to said imagechange of said eye motion detection; providing a vocal control through amicrophone and a vocal control software of said electronic computingdevice to output a second control signal through voice recognition; andcontrolling said virtual object displayed on said display in responsiveto said first, second or combination control signal without touchingsaid display of said computing device.
 2. The method of claim 1, furtherproviding infrared ray (IR) source for detecting said user.
 3. Themethod of claim 1, wherein said control signal includes cursor movement,character input, application instruction, webpage movement or image sizechange.
 4. The method of claim 1, wherein dual light sources are used toproject two spatially separated spots at user's pupil, and said duallight sources are constructed by side by side on a panel said portabledevice.
 5. A method of controlling a virtual object by eye motion andfigure motion of a user for an electronic computing device having adisplay comprising: providing at least one image sensor for said displayof said electronic computing device to capture image of said user infront of said display, wherein said at least one image sensor includesCMOS or CCD sensor, wherein said at least one image sensor provided todetect eye motion of said user to output a first control signal to avirtual object without touching said display, said user being located infront of said at least one image sensor and keeping said display with adistance; capturing images of said eye motion of said user withouttouching said display by said at least one image sensor; analyzing imagechange of said eye motion; generating a first control signal inresponsive to said image change of said eye motion detection; capturingimages of said figure motion of said user without touching said displayby said at least one image sensor; analyzing image change of said figuremotion; generating a second control signal in responsive to said imagechange of said figure motion detection; and controlling said virtualobject displayed on said display in responsive to said first, second orcombination control signal without touching said display of saidcomputing device.
 6. The method of claim 5, further providing infraredray (IR) source for detecting said user.
 7. The method of claim 5,wherein said control signal includes cursor movement, character input,application instruction, webpage movement or image size change.
 8. Themethod of claim 5, wherein dual light sources are used to project twospatially separated spots at user's pupil, and said dual light sourcesare constructed by side by side on a panel said portable device.
 9. Amethod of controlling a virtual object by eye motion and finger motionof a user for an electronic computing device having a displaycomprising: providing at least one image sensor for said display of saidelectronic computing device to capture image of said user in front ofsaid display, wherein said at least one image sensor includes CMOS orCCD sensor, wherein said at least one image sensor provided to detecteye motion of said user to output a first control signal to a virtualobject without touching said display, said user being located in frontof said at least one image sensor and keeping said display with adistance; capturing images of said eye motion of said user withouttouching said display by said at least one image sensor; analyzing imagechange of said eye motion; generating a first control signal inresponsive to said image change of said eye motion detection; capturingimages of said finger motion of said user without touching said displayby said at least one image sensor; analyzing image change of said fingermotion; generating a second control signal in responsive to said imagechange of said finger motion detection; and controlling said virtualobject displayed on said display in responsive to said first, second orcombination control signal without touching said display of saidcomputing device.
 10. The method of claim 9, further providing infraredray (IR) source for detecting said user.
 11. The method of claim 9,wherein said control signal includes cursor movement, character input,application instruction, webpage movement or image size change.
 12. Themethod of claim 9, wherein dual light sources are used to project twospatially separated spots at user's pupil, and said dual light sourcesare constructed by side by side on a panel said portable device.